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SYNOPSIS 


Evaluation of IT system is not a new term for corporate world but as far as the evaluation 
for academic institutes is concerned not much work has been done. 

The present dissertation is a step in this direction by evaluating IT system of IIT Kanpur. 
We first introduce and discuss various methods for IT evaluation, most of which are 
business oriented but can be implemented in academic institutes with modifications. We 
next propose our method for evaluating IT system of IIT Kanpur which is based on 
survey research. Target population for the survey is the students and the computer lab 
administrators. Scoring technique is used to quantify the views of the participants. 
Through this thesis, we want to provide a report which can help the institute to provide 
better IT system to support teaching, learning and research. 
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Chapter 1 


Introduction 


The amount of funds spent on IT investment has always been controversial. For many 
years it has been thought by some executives that too much has been spent and not 
enough return has been obtained from this IT expenditure. Robert Solow [15] the famous 
Nobel Prize winning economist has suggested that in business one could see computers 
everywhere but in the productivity statistics. Brynjolfsson [3] pointed out that there was a 
productivity paradox, which meant that computers were not delivering, the value 
promised. The Economist in 1991 claimed the return on IT was so poor that 
organizations: 

would have done better... to have invested that same capital in almost any other part of 
their business [5] 

1.1 Purpose of an IT System 

The purpose of the IT System is most critical to the process of defining the approach to 
its evaluation and to its performance measurement. There are a number of purposes of an 
IT System. Following are a few: 

> Improving efficiency 

> Improving effectiveness 

> Competitive leap 

> Survival 

> Inffastmcture 

1.2 Why evaluate IT scenario? 

In corporate world the evaluation of IT is at best a difficult process, which in some 
certain circumstances can be quite costly and therefore should be made clear to 
everybody why this process is being undertaken. The main reason behind the evaluation 
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of IT system is that there is considerable doubt in many circles that investment in IT has 
proved beneficial at all. It is clear that investment in IT is high therefore it is essential to 
be sure that this investment is justifiable. 

1.3 The problems with IT benefit measurement and management 

The measuring and managing of IT benefits is a difficult challenge that has plagued the 
IT industry, IT professionals, consultants and academics, for many years. The main 
reason for this is that despite the considerable amount of research conducted by 
academics and consultants so far no comprehensive or rigorous method of evaluation has 
been developed 

There have been four major areas that have contributed to the problems with IT benefit 
measurement and management. These are; 

1) Benefits and identifiable performance improvements 

2) The issues of information systems reach 

3) Tangible and intangible benefits 

4) Benefit evolution 

In the first place evaluation of IT is made difficult because of the types of benefit that 
organization can expect to derive from its application. These benefits range from 
improved efficiency to enhanced effectiveness. Both these notions of efficiency and 
effectiveness may take numerous forms and much of the task of evaluation of IT 
investment involves understanding these phenomena. 

Generally speaking benefits are divided into two basic groups, which may be described as 
tangible benefits and intangible benefits. Although tangible axe inclined to be 
straightforward to evaluate, some benefits are intangible in that they can clearly be seen 
to, for example, provide better information. However, this is difficult to measure, and it is 
even harder to directly link the end benefit to an improvement in the performance of the 
institute. 
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1.4 Objective of the Present Work 


The present dissertation is an attempt to evaluate the Information Technology System of 
IIT Kanpur. Specifically this work has the following objectives: 

> To ascertain whether the users are satisfied with the system 

> To ascertain where the users are satisfied or dissatisfied. 

> To find out what contributes to their disappointment 

> To find out how the users rate the facilities provided 

> To prioritize improvement opportunities. 

The overall goal is to provide a report which can help the institute to provide better IT 
system to support teaching, learning and research. 

Another objective is to give a new insight of the IT scenario to the institute. The 
evaluation process is in itself a learning experience. By conducting an IT evaluation in 
IIT Kanpur, the institute will learn how well it used its funds and the intellectual capital. 
This will give the institute the opportunity to understand how IT may be better employed 
and a feedback process may be put in place in hope that better decisions are made next 
time around. 

Although it is likely that some form of evaluation is constantly taking place in the IT 
environment of the institute, perhaps it is on an informal basis, with administration 
making judgments based only on its own perceptions. Indeed, sometimes these judgments 
are based on the level of complaints made about a computer system. In such cases the 
decisions of those members of staff who are most verbal or animated get to be known and 
they receive an inordinate amount of the IT support or investment resources. This usually 
regarded as being unsatisfactory. It is important, therefore, to establish a formal 
evaluation process whereby administration can make decisions on the actions most 
appropriate to the institute. 
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Therefore there is a very deep need to regularly measure performance of the institute to 
enhance performance as well as to ensure growth. Both performance and growth 
enhancement are required to meet stake holding expectations from the institute. 

1.5 What constitutes good performance? 

As far as this work is concerned we are concentrating only on IIT Kanpur. Being an 
academic institute of high repute, performance measures like reputation, academic 
excellence, user satisfaction are far more important then the monitory factors. Therefore 
we will measure the performance of the IT system based on the reputation, academic 
excellence, user satisfaction etc. only while ignoring the economical aspects. 

1.6 Organization of the Thesis 

Chapter two entitled Literature Review provides a reference for the IT evaluation during 
the last years and also gives an idea of the basis of IT evaluation. It covers the different 
techniques for the IT evaluation techniques. 

Chapter three entitled The Proposed Evaluation Technique covers in detail about our 
approach of evaluating IT system of IIT Kanpur. 

Chapter four entitled Execution and Results covers the details of data collection and 
results of the study. 

Chapter five discusses Conclusion and Suggestions. In this chapter strengths, 
weaknesses and a list of action points is given. 

Appendix I specifies Student Questionnaire 

Appendix II specifies Administration Questionnaire. 
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Chapter2 


Literature Review 


...if nothing changes about the way work is done and the role of IT is simply to automate 
an existing process, benefits are likely to be minimal. 

Thomas Davenport [4] 

The process of IT evaluation is mainly done in the corporate word. Most of the 
companies now days are getting evaluated their IT system in order to correlate it with the 
overall objective of the company. A lot of work had been done for IT evaluation as far as 
corporate word is concerned. 

Not much literature is available for such kind of evaluation being done for an academic 
institute; most of the techniques for evaluation of IT are business oriented. 

The purpose of this chapter is to review key studies from the research and the techniques 
of IT evaluation, most of which are business oriented, but still can be implemented in an 
academic institute with some modifications. 

2.1 How Well is IT System Doing? 

There is no simple answer to this question. Some organizations are content with 
establishing whether their objectives are being met? This is relatively easy, especially if 
objectives and key performance indicators were predetermined. 

However, other organizations want a more absolute evaluation. There are basically only 
two ways of approaching this type of evaluation. One, the organization can compare 
themselves to their competitors. Organization may often obtain a significant amount of 
information about the way competitors are using IT. 

The second approach to answering the question of how well the IT system is doing is to 
compare current performance with the historic performance, which requires the existence 
of a database of detailed measurements of performance of IT. This is the most 
satisfactory way of answering the question. 


5 




2.2 A Concept of Value 


The objective of this chapter is to discuss methodologies by which it will be possible to 
evaluate the benefits derived from the application of Information Technology. In 
achieving this, it will be necessary to consider inter alia, how it is possible to convert the 
benefits generated by systems based on IT into measurable values, as well as how it will 
be possible to place specific values on less tangible forms of benefits. 

Before such an exercise can be contemplated, it is essential to establish a framework in 
which the value of systems may be assessed. The shorter Oxford English Dictionary 
states that; 

Value is the amount of some commodity, medium exchange, etc., which is considered to 
be an equivalent for something else; a fair or adequate equivalent or return. [ 16 ] 

In business tenns, a rough and ready definition of value is generally accepted to be the 
amount of money that changes hands when a willing buyer trades a good or service with 
a willing seller. 

2.3 The Assessment of IT System 

The assessment or evaluation of IT, and especially IT effectiveness, is a difficult task that 
needs to be undertaken with considerable care. In addition, attention should be paid to the 
reasons why the assessment is being undertaken, as the approach to the evaluation 
depends on the purpose for which it will be used. In practice an important consideration 
is the individual who commissions the IT effectiveness study. 

If the administration is asking the question of effectiveness, then the focus is probably on 
the issue of whether the computers are enhancing the general performance of the 
organization as a whole? 
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If this question of effectiveness is being asked by the operating management, the focus of 
effectiveness is probably directed at the issue of whether the organization is getting the 
most from IT investment. 

If this question of effectiveness is being asked by the end users, the focus of effectiveness 
is probably directed at the issue of whether the IT is helping in their pursuit of excellence. 

This means that a much more general position needs to be considered as it is possible for 
a computer to enhance effectiveness and efficiency at the expense of the overall objective 
of the institute. 

2.4 Choice of Evaluation Methodology 

There is in fact a bewildering choice of evaluation methodologies available to 
management. Each methodology is designed to assess the organization’s IT effectiveness 
in a different way. The critical skill is to be able to select the methodology most 
appropriate for the organization’s particular circumstances. In the following section a 
number of evaluation techniques are given according to the application. 

2.5 Issues and Techniques for IT Evaluation 

The purpose of the investment is most critical to the process of defining the approach to 
its evaluation and to its performance measurement. As mentioned in one of the previous 
sections there are various evaluation techniques however it is a matter of type of 
application that which evaltiation technique is most appropriate. For example IT 
investment that is used to improve efficiency requires efficiency measuring techniques 
such as work study or cost benefit analysis. IT investment which has been implemented 
to enhance management effectiveness requires value added analysis, value chain 
assessment, etc. IT investment for business advantage or business transformation requires 
measuring techniques such as strategic analysis, relative competitive performance, etc. 
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2.5.1 Different Approaches to Measurement 


There are two generic approaches to measurement. These approaches are common to all 
forms of measurement, whether the measurement relates to speed, obesity, water flow, 
beauty contests, weighing potatoes or assessing information systems. 

Measurement may be based upon: 

> Physical counting 

> Assessment by ordering, ranking or scoring. 

Whether the contents of a tanker, the weight of a boxer or the speed of a jet plane are 
being measured, units are being counted. When it is difficult or not feasible to count, the 
assessment is made by ordering, ranking or scoring. [14] 

2.5.2 Specific Methodologies 

There are several different methodologies available to assess the performance of IT. The 
following are a few of the most commonly used. [14] 

1) Strategic match analysis and evaluation 

2) Value chain assessment 

3) Relative competitive performance 

4) Proportion of management vision achieved 

5) Work study assessment 

6) Economic assessment- I/O analysis 

7) User attitudes 

8) User utility assessment 

9) Value added analysis 

1 0) Return on management 

1 1) Multi-objective, multi- criteria methods 
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2.5.2. 1 Strategic Match Analysis and Evaluation 

This is a ranking and scoring technique that required the entire primary IT systems to be 
assessed in terms of whether or not they support the organization’s generic strategy. The 
two main generic strategies are differentiation and cost reduction. If a system helps 
improve customer service it generally helps the organization differentiate itself in the 
market, while if it helps to reduce costs it is generally supportive of a cost reduction 
strategy. 

2.5.2.2 Value Chain Assessment 

This is another scoring or ranking system. In this case the Michael Porter value added 
chain [20] is used as the basic checklist to which the organization’s application systems 
are compared. This process may be conducted in terms of the organization’s internal 
value activities as well as the industry value activities. For a through analysis, both 
approaches should be used. In a similar way to the strategic match analysis [2.5.2. 1], 
evaluation weights may be associated with the more important systems and scores may 
be given on the basis of the degree to which these systems have been implemented and 
are achieving objectives. 

2.5.2.3 Relative Competitive Performance 

Some organizations assess their performance by comparing themselves to their 
competitors. This requires monitoring their competitors’ acquisition of IT, the way they 
use it to achieve their goals and objectives. This is quite a difficult process and frequently 
rely on very subjective evaluations involving ranking and scoring. 
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1.5.2 A Proportion of Management Vision Achieved 


This is another ranking and scoring technique that has a high degree of subjectivity. 
Managers are asked to assess the current system in terms of what their original plans 
were. When a large number of managers are involved a questionnaire or survey approach 
may be used. Despite being subjective, this approach can be applied in a relatively 
objective way by conducting regular assessments on a six or twelve- monthly basis. 

2.5.2.5 Work Study Assessment 

The work study approach to IT benefit evaluation requires regular reviews of how the 
work in the department is being performed. During these reviews the volume of work is 
carefully recorded as well as the time required to perform all the necessary tasks. Work 
study assessment can be relatively objective particularly if work study professionals 
conduct it. The results of a work study appraisal may be used as input to subsequent cost 
benefit analysis. 

2.5.2.6 Economic Assessment 

An economic assessment is a theoretical approach to IT benefit evaluation. It requires the 
development of a model expressed in mathematical terms in which the relationship of 
input and output are expressed. Although applying rigorous mathematical terms this 
method also relies on subjective views of the nature of the relationships between the input 
and output variables. 

1.5.2.1 User Attitudes 

User attitudes may be used to assess how IT is performing within the organization. In 
this, a survey method is used to extract attitudes towards the importance of IT to the 
individual users, as well as how the administration is performing in its delivery of IT. 
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2.5.2.8 User Utility Assessment 


It may be argued that systems that are heavily used are more successful than those that 
are not. By establishing the frequency of use of a system it is believed that it is possible 
to assess its value to the organization. This technique involves counting the amount of 
activity sustained by the system, measured in terms of its input, processing and output. 

However, excellent systems have been known to fail for the trivial reasons, while poor 
systems have survived for years and therefore this approach leaves many questions 
unanswered. 

2.5.2.9 Value Added Analysis 

Using this technique the value of the system rather than its cost is firstly assessed. Once 
the value or benefit has been agreed then a cost is calculated. The system is then 
developed on a prototyping basis as a relatively low cost. Once this has been completed 
an assessment is made to decide whether the benefits derived have justified the cost. If 
the decision is that the investment is justified then the organization proceeds to the next 
stage in the IT applications development. 

2.5.2.10 Return on Management 

Return on management is a concept proposed by Paul Strassmann initially in his book 
Information Payoff (Strassmann, 1985) [16] and again in The Business Value of 
Computers (Strassmaim, 1990) [17]. The return on management (ROM) method is a 
valued added approach that isolates the management added value and then divides this by 
the management cost. 

When expressed as a formula; 

Management value added = ROM / management cost 
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Management value added is the residue after every contribution to an organization’s 
inputs is paid. If management value added is greater than management cost then 
management efforts are productive in the sense that outputs exceeds inputs. 


2.5.2.11 Multi-Objective Multi- Criteria Methods 

Multi-objective multi- criteria methods are subjective methods for appraising the value of 
different outcomes in terms of the decision makers own preferences. It assumes that the 
value of a project or IT investment may be determined or measured in terms other then 
money. Therefore, preferences are used instead of ROI, etc. 

The philosophy behind multi-objective multi- criteria methods is that different 
stakeholders may have different ideas about the value of different aspects of an IT 
investment and that these may not be strictly comparable in money terms. This technique 
allows different views and values to be investigated and assists in exposing potential 
conflicts at the decision making stage. 

Multi-objective multi- criteria methods are most useful when strategic choices have to be 
made between quite different alternatives. 

2.5.3 Classification of Methodologies 

Each of these methodologies leads to the development of a measure or metric that allows 
the IT to be evaluated. Sometimes the metric is compared to a standard or sometimes 
relative metrics are compared, for competing, systems. However, in most cases a single 
measure is not sufficient to make an evaluation. Two or three metrics will usually be 
required but six or seven measures should be avoided. These methodologies may be 
categorized as primarily objective or subjective in nature. However, even in more 
objectively oriented approaches the calculation of the metric will almost invariably be 
based on subjective criteria and any suggestion that the method is totally objective should 
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be resisted. Fig. 2.1 categorizes these measures in terms of their relative subjectivity/ 

objectivity. [14] 

Classification Evaluation Approach 

Partially Objective Cost benefit analysis 

Economic analysis 

System usage 

User utility assessment 

Relative competitive performance 

Work study assessment 

Value added analysis 

Fully subjective User attitudes 

Management vision 
Value chain assessment 
, , Stmteg|c;matQh analysis 
Table 2.1 Classification of Evaluation Methodologies 

2.5.4 Limitations of these Methodologies 

Ail approaches to IT benefit assessment have significant conceptual or practical flaws. 

This does not mean that they can not be or should not be used, but practitioners should be 

aware of their limitations. [14] 

2.5.4.1 Strategic Match Analysis and Evaluation 

1) Highly subjective 

2) Issues not well understood 

3) All but top management may be unaware of strategy 


13 



2.5.4.2 Value Chain Assessment 


1) Very subjective 

2) Difficult to obtain hard data 

3) Not well understood by management 

2.5.43 Relative Competitive Performance 

1) Information available may be sketchy 

2) Difficult to compare benefits of different systems 

3) Uncertainty about competitors’ plans 

2.5.4.4 Proportion of Management Vision Achieved 

1) No hard data 

2) Virtually no objectivity in this approach to assessment 

3) Not always easy to get top management to admit to failure 

2.5.4-5 Work Study Assessment 

1) Objectivity may be relatively superficial 

2) Changes in work patterns may drastically alter the assessment 

3) Most managers are not familiar with these techniques 

2.5.4.6 Economic Assessment- I/O Analysis 

1) Requires an understanding of economic analysis 

2) Relatively abstract 

3) Attempts to avoid detailed quantification of monetary terms 

4) Most managers are not familiar with these techniques 
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2.5.4.7 User Attitudes and User Utility Assessment 

1) A very technical approach 

2) Not many practitioners available 

3) Has only a low relevance to operating staff 

4) Users may not tell the truth or may simply exaggerate. 

5) Users may have vested interests in presenting a particular viewpoint 

6) Corporate culture may color users’ views and the interpretation of the outcome 

2.5.4.8 Value Added Analysis 

1) Very practical approach 

2) Keeps costs under control 

3) Encourages prototyping 

2.5.4.9 Return on Management 

1 ) A major break with classical economics 

2) Not easy to operationalise 

3) Useful to stimulate re-thinking 

2.5.4.10 Multi-Objective, Multi- Criteria Methods 

1) An unquantifiable method 

2) Not useful as a post implementation tool 

3) Useful to stimulate debate 


15 



2.5.5 A Holistic Approach to IT System Evaluation 


The IT function of an organization is involved in the development, implementation and 
maintenance of numerous systems. These systems aim to meet needs at all levels within 
the organization. In evaluating the performance of the IT, it is normally necessary to 
evaluate the performance of the individual systems, and then use the aggregate of the 
performances on the individual systems as an overall measure of the success of the IT 
function. 

Measurement of IT effectiveness is a key issue; unfortunately, there is little agreement on 
how to measure effectiveness. 

The measurement problem is exacerbated by many ways in which effectiveness may be 
viewed. For an example a particular IT system can be considered effective when it: 

> Is meeting its objectives 

> Operates within its budgets 

> Delivers on time 

> Is a major catalyst in directing the organization’s use of it 

> Ensures that the organization is using it competitively 

> Has a clear understood role in the system 

> In generally perceived to be an ally 

> Is perceived by administration to be value for money and users believe that IT is 
being deployed in a way which supports their pursuit of excellence. 

2.5.5. 1 Goal Centred vs. Systems’ Resources [8, 9] 

There are basically two general views with respect to measuring IT effectiveness. These 
are 

> Goal centred view 

> Systems’ resource view 
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I n the Goal centred view we focus on the out comes of the IT function. We determine the 
task objectives of the system and then establish the criteria for measuring whether these 
objectives have been achieved. 

In the systems’ resources view we focus on the process or the functional aspects of the 
system. In this case effectiveness is measured against such things as user satisfaction, 
communication between IT staff and users, and quality of service. 

There are a number of ways to measure the effectiveness of IT system. User information 
satisfaction (UIS) [11, 12] is recognized as an important indicator of IT effectiveness. It 
is on this approach to the measurement problem that we now concentrate, and in 
particular on perceptual measures of UIS. This involves incorporating user feelings, 
beliefs, opinions and attitudes toward IT into the evaluation procedure. 

In the context of IT effectiveness, it is generally believed that if users declare themselves 
to be satisfied with the system then the system may be said effective. Clearly, such 
satisfaction measurement is at best an indirect and relative measure, which must be used 
with considerable care. It may be the case that users could be happy with inadequate 
system. 

2.5.5.2 User Information Satisfaction (UIS) [11, 12] 

User satisfaction is generally considered to result from a comparison of user expectations 
(or needs) of the IT system with the perceived performance (or capability) of the IT 
system on a number of different facets of the IT system. This is considered to be a 
holistic approach to systems effectiveness as it addresses the whole IT function rather 
than individual systems. 

More specifically, overall attitude to the IT function can be considered to be influenced 
by the size and direction of the discrepancies (or gaps) between expectations and 
performance. A positive (negative) gap results when perceived performance exceeds (is 
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below) expectation. A large ‘positive’ gap can be interpreted as indicating that IT 
resources are being wasted, whereas a large ‘negative’ gap indicates a need for improved 
performance. 

A variant to the above approach is to use the correlation between expectations and 
performance scores as a measure of ‘fit’. The correlations provide a means for assessing 
the overall effectiveness of the IT function, where high positive correlations can be taken 
to imply ‘consensus’ of views. 

Of the many published papers on UIS two will be discussed in more detail below, 
namely, the Miller and Doyle (1987) [12], and the Kim (1990) [11] papers. Both these 
studies propose conceptual models to explain UIS, thereby adding credence to the 
instruments developed ftom them. These models have their roots in the theory of 
organizational and consumer behavior. This is easy to comprehend if one accepts that the 
IT function impacts on the whole system and is aimed at satisfying the user. 

2.5.5.3 A Gap Approach to Measurement 
2.5.5.3.1 The Kim Model [11] 

A feature of Kim’s model is that UIS is considered to be influenced not only by post 
implementation experience of the IT system. The later is captured through the user’s 
initial expectations of the IT system. 

In this approach, UIS is measured by the discrepancy between the user’s perception score 
of the IT performance and the user’s expectation score of the IT. Furflier, the model 
describes how UIS is influenced by the discrepancies that arise during the developmental 
and service delivery processes. The developmental stage comprises two sub stages, 
namely, the determination of the IT requirements and the design and installation of the IT 
system. These various stages give rise to three gaps that influence the UIS. These gaps in 
turn may influenced by various factors. Examples include: user participation in defining 
the IT requirements; administration support, and the user training which is likely to 
impact on the gap between the actual quality of the IT system installed and the what user 
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perceives the quality to be through use of the system. There may be other factors which 
directly impact on the IT system, rather than indirectly through the gaps. The Kim model 
is represented diagrammatically in figure. 2.1 



Figure 2.1 The Kim Model 


Interpretation of the gaps 

Gap 1 this is the discrepancy between the users’ expectations of the IT and the systems 
designers’ interpretations of these expectations. 

Gap 2 this is the discrepancy between the IT specialist’s interpretation of the users’ needs 
and the quality of what is actually installed for the user. 

Gap 3 this is the discrepancy between the quality of what is actually installed and what 
the user experiences when interacting with IT system. 


19 











Incorporation of organizational factors 


The model postulates that the three gaps can be affected by organizational factors. For 
example, Gap 1 could be influenced in a positive way by encouraging user participation 
in the design stage. This involves determining the information requirements desired from 
the IT system. On the other hand, administration support for IT, exhibited through, say, 
the provision of enough resources should be positively correlated with Gap 2. Finally the 
provision of proper training should be positively correlated with Gap3. 

Formulating and fitting the model 

UIS is measured as the discrepancy between user expectations and the perceptions of the 
IT system. Furthermore, the model assumes that overall UIS can be explained by Gaps 1 
to 3 and also organizational factors. More formally: 

UIS = f (Gap 1, Gap 2, Gap 3, Organizational factors) 

To operationalise the model, it will be necessary in the first instance to develop 
instruments to measure three gaps. This should be possible by applying the 
methodologies used by researchers when modeling consumer satisfaction with quality of 
service, where consumer satisfaction is expressed as a function of a number of gaps 
[Parasuraman et al. 1985, 1988; Brown and Swartz, 1989]. The approach most used to 
conceptualize and determine the dimensions for the evaluation of the IT effectiveness is 
multivariate statistical analysis technique of factor analysis. Once these instruments are 
available it should be possible, through the use of correlation and regression analysis, to 
determine which organizational factors affect these gaps. Also, the extent and nature of 
the influence of the gaps and the identified organizational factors on overall UIS can be 
determined through the use of such statistical methods. 
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2.5.5.3.2 The Miller- Doyle Approach [12] 


Of the many instruments proposed for measuring, through perception, user satisfaction 
with information systems, the one due to Miller and Doyle is described here ( Miller and 
Doyle, 1987). It is, in spirit, similar to the conceptual model described above. The 
instrument has been extensively used in many different firms, in many different sectors, 
and the results provide convincing evidence of the instrument’s reliability and validity. 

Description of the instrument 

The instrument is designed to measure the perceived effectiveness of the overall IT 
function and involves the use of a questionnaire. The questionnaire comprises five parts, 
AtoE. 

Part A consists of the questions which measure the extent to which certain facets of the 
IT system are perceived to be important in ensuring the organization’s IT system is 
effective and successful. The attitudes are rated on a semantic differential scale of 
1 (irrelevant) to 7(very critical). 

Part B consists of the questions on the future needs for IT system. 

Part C consists of the same questions as Part A but in the case the respondent is asked to 
rate the questions with respect to the actual performance achieved. Again a seven point 
scale is used in this case the levels of perceived importance go from l(very poor) to 
7(excellent). 

Part D consists of the questions relating to the organization’s performance in developing 
new systems. 

Part E consists of questions which capture certain demographic data. There is also a 
question which asks for a rating of the organization’s overall IT performance on a scale 
of 1 (complete failure) to 7 (very successful). 

The important ratings in Parts A and B capture perceptions on the business needs, while 
the performance ratings in Part C and D capture perceptions of the organization’s IT 
capabilities. 
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A factor analysis of these ratings revealed that there were seven dimensions of user 
satisfaction underlying the responses to these questions. These are: 

> Functioning of existing transaction/ reporting systems. 

> Linkage to strategic processes of die firm. 

> Amount and quality of the user involved. 

> Responsiveness to new systems needs. 

> Ability to respond to end user computing needs 

> IT staff quality 

> Reliability of service. 

Interpreting the results 

The mean of the performance responses to each of the question can be taken as the 
measure of the perceived performance on each of the facets. 

Overall user attitude to the IT function is measured by a composite score derived from 
the user performance mean ratings on the 38 questions, by calculating their mean value. 
This gives the user an overall assessment of the organizations IT capabilities. 

The fit between importance and performance ratings can be measured by the square of 
the correlations between these scales, and/ or by the discrepancies (gaps) between these 
scales. Miller and Doyle recommend the use of the correlation as the preferred measure 
of success of the IT system. 
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Chapter 3 


The Proposed Evaluation Technique 


3.1 Efficiency and Effectiveness Studies 

The proposed model is based on a method [14] which is slightly different, but relatively 
close to measuring IT benefits, is that of conducting Value of the IT System (VITS) 
study. 

The aim of the VITS study is to establish whether the IT system of IIT Kanpur is 
functioning efficiently and effectively. 

3.2 VITS 

The Value of the IT System (VITS) study is aimed at highlighting areas of excellence, as 
identifying those areas requiring remedial action. The study includes a discussion of the 
various areas in which an IT system may be improved, and the selection of a practical 
way of so doing with relation to the institute. 

The deliverables of VITS study inter alia include: 

1) A balanced report presented as objectively as possible highlighting the strengths 
and weaknesses of the IT system; 

2) A better idea of how to use the resources available to IT system; 

3) A more motivated IT system management team who, as a result of participating in 
the study, have been highly involved in assessing their own work and that of their 
colleagues; 

4) A list of action points. 
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3.3 VITS of IITK: Setting up the Study 


First step is to define and clarify objectives and the scope of the study in the direct 
relation to the institute; 

Second step is to design a questionnaire or/ and interview for the survey according to the 
objectives and scope of the study; 

Third step is to define the target sample which can answer the survey questions; 

Fourth step is to execute the exercise of collecting data through questionnaire or/ and by 
interviews; 

Fifth step is to analyze the data and finally; 

Sixth step is to conclude and give suggestions for improvements. 

3.3.1 Defining Objectives and Scope of the Study: Step 1 

The precise scope of the study categorizes will vary extensively depending on the 
organization. For IIT Kanpur the following eleven identifiable areas are considered for 
setting up the study: 

1) Awareness 

2) Training 

3) Time spent in learning 

4) Hardware 

5) Software 

6) Internet and Networking 

7) Utilization 

8) Staffing 
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9) Service levels 

10) Security 

1 1) User support 

3.3.1. 1. Awareness 

There is no use of any facility if the users do not know what they can avail. Awareness is 
a key area of the IT system. Awareness takes into account both users and administration’s 
point of view like if administration is doing some thing to make users aware or not and 
also whether users are willing to know what facilities they are being provided. 

3.3.1.2 Training 

Training is an equally important area as awareness is. Main advantage of training is it 
saves time of the users. The commitment to training from the IT system will be examined 
as efficiency can be greatly enhanced if users are properly trained in applications they 
use. 

3.3.1. 3 Time spent in learning 

It is important to know how much time users are spending only on the learning part. As 
far as IIT Kanpur is concerned time of our intellectual capital that is students, is far 
important then any other factor like economics of the IT system. So it is important to 
check whether students are spending more time in learning only or spending more time in 
application part. 

3.3.1.4 Hardware and software 

It is important to determine exactly what hardware and software exists in the institute, 
and what is in use in the institute. This will involve personal computers on the desks of 



end users, as well as the equipments like scanner, printer plotter etc. The range of 
hardware configurations and versions of software in use is to be established. 
Establishment of a hardware and software asset register is a key component in any VITS 
study. Other aspects of the hardware and software inventory include what software is 
installed on machines, but not actually used by the user? What is the future plan for 
hardware and software growth? 

The areas of hardware and software maintenance will be addressed. How much of this is 
outsourced and how much is performed internally? 

3.3.1. 5 Networks 

This area will cover the issues like the current design and configuration of online 
networks in terms of efficiency. To find out whether networks have enough capacity and 
are performing to an efficient level. What is the loading and the downtime of the network 
and can this be improved? How many user connections are their currently and can this be 
increased? 

3.3.1.6 Utilization 

This area covers two very important factors of the IT system. One is miss use and the 
other one is under utilization. Both are equally bad for any of the system. 

3.3.1. 7 Staffing and Service Levels 

This involves looking at the current agreements and commitments that the IT system has 
to users in terms of turnaround times, availability, response times, etc. If a service or help 
desk exists, its efficiency in terms of staffing and response time will be examined. What 
tools for recording problems and solutions are used? 



3.3.1. 8 Security 


The study examines the computer related security like controls over input, processing, 
and output procedures. Special attention to the control of operating software will be 
given. What, if any, written procedures are there, and what happens in actual practice? 
How does the institute intend to minimize attacks from hackers and virases? 

3.3.1.9 User Support 

How much support do users feel they get from IT system and how approachable do they 
feel IT system personnel are? Are requests dealt with promptly and sympathetically? 

3.3.2 Designing a Survey: Step 2 
3.3.2.1 What is a Survey? 

A survey is a procedure for collecting information from individuals. The information 
sought can range from general background information to that relating to the beliefs, 
feelings, attitudes, life styles and intentions, etc. methods for obtaining such information 
usually fall in two categories, namely self completion and interview administered. Self 
completion methods include mail surveys, computerized surveys, and on board or in 
company surveys. Interview administered surveys include personal interviews, telephone, 
and on board surveys. 

Each method possesses advantages and disadvantages. They do, however, all depend on 
the basic assumption that individuals subjected to questioning will be both willing and 
able to give truthful answers to the questions posed. [14] 

In the present work both the methods are used. 

Questionnaires of the Survey for our work are given in appendix I and II. 



3.3.3 Defining Sample Type and Size and Data Collection: Step 3 and 4 


Sample type means to whom the survey should be targeted. In our work target people are 
students and the IT lab administrators. Given the greater number of responses from the 
random sample for the student community there is 95 % confidence level [4.2.5]. Details 
of the student survey sample size and information regarding data collection are given in 
the next chapter. Administrative survey is qualitative in nature. 

3.3.4 Analyzing the Results: Step 5 

The performance of the IT system in each of the categories is assessed on a scale of four 
points. The Scale has categories: very good, good, average, poor and very poor with 
average being 2.5. The scores are derived by averaging views obtained at interviews and 
by collecting data using a questionnaire. 

The results of the scores are presented in a number of different ways. One method of 
presentation, used by Coopers & Lybrand Deloitte, is that of wheel diagram. Another is a 
bar chart. 

The scaling system and the analysis of results are given in detail in the next chapter. 

3.3.5 Conclusion and Suggestions: Step 6 

Based on the analysis, the VITS study indicates the current performance of the IT system 
of IIT Kanpur by mentioning strengths and weaknesses and how the performance may be 
improved. This is expressed through a list of action points according to the priority. 
Conclusion and suggestions are given in Chapter 5. 
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Chapter 4 


Execution and Results 


4.1 The Survey 


The survey was addressed to students and computer lab administrators. Two 
questionnaires were designed one each for students and administrators. Student survey 
questionnaire is given in appendix I and administrator survey questionnaire is given in 
appendix II. In both the surveys respondent were chosen randomly. 

4.2 Student Survey 
4.2.1 Survey Population 


Student survey was of on board type. Hard copies of the questiormaire were distributed 
randomly in CC, Hall 1, Hall 2, Hall 3, Hall 4, Hall 5 and GH.600 copies were distributed 
out of which 564 people responded. 

In the table 4. 1 the statistics of the survey are given. 




Number oti 

Number of students responded 564 

Number of UG students tJshorespoSiSea'', ,>571- ' 

Number of PG students who responded 228 

Number of students who did not miOTtidn feeir prograutn y- 

Table 4. 1 Student Survey Population Statistics 
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Figure 4.1 and 4.2 shows the distribution of the survey population. 

As the figure 4.1 shows that population contains 30% UG students, 41% PG students and 
29% students did not specify their program. 

Figure 4.2 shows that survey covered students of almost every branch or program running 
in IIT Kanpur. 


'^f\ 




SURVEY POPULATION- PROGRAM 


UNSPECIFIED 



PG. 228, 41% 


Figure 4. 1 


SURVEY POPULATION- BRANCH 


MATHS 



Figure 4.2 


4.2,2 Rating System 


Questions of the survey are grouped in eleven areas around which this study revolves. 
Weights have been assigned to the answers of each question according to their 
importance. Assignment of weights is a very subjective exercise and is a result of 
previous experience sund brainstorming sessions. The objective of assigning weights to 
each alternative is to quantify the responses so that the IT system can be evaluated more 
effectively. 

Some questions have 4 alternatives and some have 3 alternatives. Table 4.2 shows the 


RANGE 

0.0 <= X < 2.0 
2.0<=X<2.5 

2.5 <= X < 3.0 
3.0<=X<3.5 

3.5 <=X<4.0 

Where, X is the overall mean of a particular area. 

Table 4.2 Ratings 

range of weights and their corresponding rating. Based on these ratings weights have 
been assigned. 4 points orl point is assigned in extreme cases and 2.5 point is assigned in 
average case if there are three options and 4, 3, 2, 1 are assigned if there are four options. 

Following is the list of areas with questions and weights assigned to their answers: 

4.2.2.1 Awareness 


RATING 

VERY POOR 
POOR 
AVERAGE 
GOOD 

VERY GOOD 


1) Are you aware how many software/ packages do you have in your lab or in CC? 


a) Fully aware 4 

b) Aware about whatever is required 3 

c) Not concerned 2 

d) Not Aware 1 


2) Was there any orientation program held in the beginning for the awareness of IT facilities 


available in IIT Kanpur or at least in your lab? 

a) Yes 4 

b) Yes, there was a program but it was not sufficient 3 

c) Not awsire 2 

d) No 1 
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4.2.2.2 Training 


Is there any training facility available for the software you worked on? 


a) Yes 4 

b) Yes but not sufficient 3 

c) Not Aware 2 

d) No 1 


4.2.23 Time Spent 

If the training facility was not available then what was the estimated fraction of time spent on non 
productive work like figuring out which software to use, how to access it and how to use it? 


a) < 25 percent 4 

b) Between 25 percent to 50 percent 3 

c) Between 50 percent to 75 percent 2 

d) > 75 percent 1 


4.2.2.4 Hardware 

Are hardware facilities available in IIT Kanpur sufficient? 


a) Yes 4 

b) Yes, but there is a room for improvement 3 

c) No 2 

d) Pathetic 1 


4.2.2.5 Software 

Are software/packages available in IIT Kanpur sufficient? 


a) Yes 4 

b) Y es, but there is a room for improvement 3 

c) No 2 

d) Pathetic 1 



4.2.2.6 Internet and Networking 


Are net connection and other networking facilities available in IIT Kanpur sufficient? 


a) Yes 4 

b) Yes, but there is a room for improvement 3 

c) No 2 

d) Pathetic 1 


4.2.2.7 Utilization 


Are students misusing the IT facilities available in UT Kanpur? 

a) Yes 1 

b) Can’t say 2.5 

c) No 4 

Are students under utilizing the IT facilities available in IIT Kanpur? 

a) Yes 1 

b) Can’t say 2.5 

c) No 4 


4.2.2.8 Services 


Have you ever complained to the concerned person regarding the unavailability or inaccessibility 
of a particular software? 

a) Yes, action was taken for the complaint 4 

b) No such case was found 2.5 

c) Y es, but no action was taken 1 



4.2.2.9 Staff 


Do you find the supporting staff: 


1 ) Technically capable 

a) Yes 4 

b) Average 2.5 

c) No 1 

2) Approachable 

a) Yes 4 

b) Not always 2.5 

c) No 1 

3) Helping 

a) Yes 4 

b) Not always 2.5 

c) No 1 

4.2.2.10 Security 

Do you find the IT System secure? 

a) Absolutely secure 4 

b) Reasonably secure 3 

c) Not secure 2 

d) Dangerously vulnerable 1 

4.2.2.11 User Support 

1) Is there any online/offiine help available for the availability, accessibility of software 
packages? 

a) Yes 4 

b) Can’t say 2.5 

c) No 1 



2) Were extra needs like more hard disc space, parallel computing, use of plotter etc taken care of 
whenever required? 

a) Yes 4 

b) Can’t say 2.5 

c) No 1 

4.2.3 Summary of Results 

In figure 4.3 mean scores of each area are shown by the help of the bar chart and in Table 

4.3 sunnmary of the results is provided. 



Figure 4.3 

I 


O C 



SR.NO. 

AREA OF STUDY | 

SUBAREA 

MEAN 

MEAN 

RATING 

1 

AWARENESS 


2.72 

AVERAGE 


Awareness to s/w, h/w available 

3.03 




Orientation program for 

awareness 

2.42 



2 

TRAINING 

1.47 

1.47 

VERY POOR 

3 

TIME SPENT 

3.13 

3.13 

GOOD 

4 

HARDWARE 

3.42 

3.42 i 

GOOD 

5 

SOFTWARE , 

3.56 

3.56 1 

VERY GOOD 

6 

NETWORKS i 

3.59 

3.59 , 

! 

VERY GOOD 

7 

{ 

1 UTILIZATION I 

Miss use * 

1 

1.95 

1.95 j 

! 

VERY POOR 


Under utilization ' 

t 

1.95 



8 

SERVICES 

2.53 

2.53 , 

AVERAGE 

9 

STAFF 1 

1 


3.03 ’ 

GOOD 


Technical Capability 

! 3.19 

1 



Approachable 

J 

1 2.88 

, 



Helping | 

- 3.03 

* 


10 

SECURITY 

1 2.61 

2.61 . 

AVERAGE 

11 

USER SUPPORT 

j 

2.69 ; 

AVERAGE 

Availability of online/offline 
help 

2.71 

1 

1 ; 
f 


Extra needs 

2.67 ! 


OVERALL MEAN 

1 

2.79 ; 

AVERAGE 


„ ! 


STANBARI, DEVIATION j 0.661 i 

Table 4.3 Summary of Results: Student Survey 



As the results shows, in almost half of the areas system is doing just average. In 
networking and software areas system is doing a very good job. 

IT system of I IT Kanpur is doing very badly in providing training to their users and 
utilization of the system; these areas have managed just 1.47 and 1.95 points respectively 
with a rating of ‘very poor’. Overall the IT system has got 2.79 points which is rated as 
‘average’. 

Results are discussed in detail in the next section. 


4.2.4 Wheel Diagram 

The results of the study arc presented here as a wheel diagram [Coopers & Lybrand 
Deltoitte]. Wheel diagram is as shown in figure 4.4. This concept has been adopted to 
accommodate the eleven scope categories using the ratings given in table 4.3. The area of 
the polygon produced by joining the scores is a reflection of the VITS the institute 
receives. If the area is, the VITS is considerable, whereas if the area is small the VITS is 
low. 



UTLIZATiON 

Figure 4.4 


It is evident from the wheel diagram that the area of the polygon is not very large. It 
means that the VITS is neither considerable nor low, its average. Area left out of the 
result line gives an idea about the room for improvement in the respective areas. 


4.2.5 Confidence Interval 

The (1- a)* 1 00% confidence interval estimate for the mean with population mean 
unknown is expressed as follows 


(a-) 

Where tn-i is the critical value of the t distribution with n-1 degrees of freedom for an area 
of ot/2 in the upper tail, (z) is the sample mean and S the sample standard deviation is 
the estimate of population standard deviation. For the sample size greater than 120, S 
estimates precisely enough the population standard deviation. [1] 

For our sample 
(J) = 2.79 
S = 0.661 
n =564 

For 95% confidence level 
tses = too = 1 .9600 

We would conclude with 95% confidence that the mean score of the IT system of IIT 
Kanpur is between 2.79 ± 0.055 i.e. 

2.735 < |Jx < 2.845 

The 95% confidence interval states that we are 95% sure that the population means JJx is 

located within the interval 2.735'and 2.845. In other words, if all possible samples of size 
564 were selected (something that would never be done in practice), 95% of the intervals 
developed would include the true population mean somewhere within the interval. 



4.2.6 Detailed Results and Discussion 


In this section results are shown in full details. Details are given of each and every 
question ot the questionnaire. In the last, comments given by the students for the open 
ended question are given though some comments axe placed in between at the appropriate 
place. Responses to eveiy question are depicted by three pie charts, one of responses 
given by UG students, one is of responses given by PG students and the third one is 
combined (including those who did not mention their program). 

"Three years mv were 6'^ in IT facilities in Asia but now we are 19'^. This shows that 
people around are advancing faster than we are. IITK has been known for it’s computing 
facilities thus a humble request to improve the facilities to make them one of the best in 
Asia" 

— Student comment 

4.2.6. 1 Awareness 

OVERAI.L POINTS 2.72 

OVER AI .1 . RATING AVERAGE 

1.1) Are you aware how many software/ packages do you have in your lab or in CC? 

a) Fully aware 

b) Not concerned 

c) Aware about whatever is required 

d) Not Aware 

POINTS 3.03 

RATING GOOD 
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Figure 4.5 


Figure 4.6 
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Figure 4.7 

A.S the above charts demonstrate most of the students are aware only about whatever is 
required. Very less number of students .said that they are fully aware of the IT facilities. 

"'The facUitwx ami this .survey all Ls in vain if the users are not aware of whatever is being 
provided. To get better output students should be made aware first about how to use 
facilities. ” -- Student comment 

1.2) Was there any orientation program held in the beginning for the awareness of IT 
facilities available in IIT K^pur or at least in your lab? 

a) Yes 

b) Yes, there was a program but it was not sufficient 

c) Not aware 

d) No 

POINTS 2.42 

RATING POOR 
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ORIENTATION PROGRAIWI FOR AWARENESS 
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Figure 4.10 


In tliis section UCi students slightly disagree to each other. UG says that there was an 
orientation program while FG says no there was not. Reason may be that most of the UG 
students arc CC users while majority of the PG students use their departmental labs. 
Overall opinion is that there is no orientation program for the awareness of the IT 
facilities in 111' Kanpur. 


4.2.6.2 Training 
POINTS 1.47 

RATING VERY POOR 


Is there any training facility available for the software you worked on? 
a) Yes 


bn) Yes but not MjfTicicnt 
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Figure 4.11 


Figure 4.12 
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Figure 4.13 


J\s the above charts demonstrate population as large as 75% says that there is no training 
racility available for the software etc. they work on. Point here to appreciate is that there 
i s also an option of “training not r«|uired” but only 9% says that training is not required, 
t 1 1 means that majority says that training is required but it is not being provided. 

-A person who gets facilities like this should feelmore then content. Of course it would 

*%7ye appreciable if some body will teach working on certain software. ” 

— Student comment 

, 1 1 1 is indeed a major area of conceto and undoubtedly ttie most ignored area among all the 
Jt facilities provided. 
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4.2.6.3 Time Spent 
POiN IS ' 1 ' 

RATINCl tHHU) 


If the training facilil) \\a.s not available then what was the estimated fraction of time 
spent on non proilucli\e work like figuring out which software to use, how to access it 
and how to use it? 


a) < 25 percent 

b) Between 25 percent to 5t) percent 

c) Betwecti 50 {Krcettt to 75 percent 

d) > 75 percent 


ESTIWATeO TIME Of LEAftMINO#Q 


ESTMttTED 71116 FOR LEARMNG4X3 


f? m 

<10% 



f«) 1$ IhiFf '' ‘ 


6m 



0.32% 


« a) <25 percent 

1 b) Between 25 percent 
to 50 percent 

□ c) Between 50 percent 
to 75 percent 
Ocl)>7$ 


Figure 4.14 


Figure 4.15 


TIME SPENT 


c> B«itw»«©ri 50 
pBfcent to 75 
poremnt 
24% 



b) Botwoon 25 
percent to 50 
percant 
30% 


Figure 4.16 

Statistics shows that a good population of students spends 25% to 50 % time only in 
learning the software. Can this time be reduced by providing training? 


4.2.6.4 Harclwart* 

POINTS 5 4 ; 

RATlNti iitHM) 

Are harilv\arc facilities av ailablc in H I Kanpur sufficient? 

a) li'es 

b) Yes. but thcie is a uumi lor iinprovcincnt 

c) No 

d) Pathetic 
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Figure 4. 18 


HARDWARE FACILITIES 


C) Yen, but 

rnmm IS m room d) Pstbotic 

for innpfavemant 0% 


Z2^A* 




a) Yes 
55^ 


b> No 
13% 


Figure 4 , 1 9 


Most of the student.s say tltat hardware facilities are good. But a different point was put 
by some users in the open ended question. 


"Fa^' ^ Ktiftpur luivv hc.si possihiv nuichine.s and other facilities whereas some 

labs do not haw' even a single /’< ' ' 


— SludciU comment 

Well no doubt that the hard\\are tacilities are good as the majority says but facilities are 
not consistent through out the 1! f. 

4.2.6.5 Sofhrari* 

POINTS '.5<^ 

RATIWt MRYtiOOn 


Are software packages a\ atlahle in HT Ktmpur sufficient? 

a) Yes 

b) No 

c) Yes, but there is a room for improvement 

d) Pathetic 



Figure 4.20 
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i'igure 4.22 

As the iihosx chart.s depict that most of the users are very much satisfied with the 
software facilities. 


'■‘■ITfacUitU'x prtmtii’ti m KIT Kmpur tire best possible. ” 


- Student comment 


4.2.6.6 Internet and Netw orking 
POINTS 

RATING VI-RY (KK)l) 


Are net connection and other networking facilities available in IIT Kanpur sufficient? 

a) Yes 

b) No 

c) Yes, but there is a room for improvement 

d) Pathetic 

mnaHET and Netl«0«KIN<M»0 internet and networkingog 



li WNe 
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ttfom Kur 


d.0% 
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Figure 4.24 
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Figure 4.25 


Most of the students are satisfied with the internet and networking facilities. This facility 
is rated as “very good” along with the software facilities. 


4.2.6.7 Utilization 

OVERALL POINTS 1.95 

OVERALL RATING POOR 


6.1) Are students misusing the IT facilities available in IIT Kanpur? 

a) Yes 
c)No 

b) Can’t say 


POINTS 1.95 
RATING POOR 



Figure 4.26 


Figure 4.27 
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MISS USE 


c) Can’t say 
20 % 


a) Yes 
58 % 


Figure 4.28 

This can be appreciated that students by themselves realize that the facilities are being 
heavily miss used. Obviously it is not good for the system. 

"Abuse of internet should be checked, especially in the night” 

— Student comment 

“It ’s disgusting to see that when one really needs a PC for some urgent assignment and 
PC is not available because some body else is playing Games ” 

-- Student comment 

6.2) Are students under utilizing the IT facilities available in IIT Kanpur? 

a) Yes 

b) Can’t say 

c) No 

POINTS 1.95 

RATING POOR 
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Figure 4.29 


Figure 4.30 


UNDER UTILIZATION 


c> Can’t say 






a) Yes 
SAO/o 


Figure 4.31 

Almost the same percentage of students who says that system is being miss used, also 
says that system is heavily under utilized. Does it means that first the students are not 
using the system properly and if using then most of them are doing unproductive work? 


“Students are not using even the 50% of the facilities’’ 

— Student comment 


4.2.6.8 Services 
POINTS 2.53 

RATING AVERAGE 

Have you ever complained to the concerned person regarding the unavailability or 
inaccessibility of a particular software? 

a) No such case was found 

b) Yes, but no action was taken 
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c) \'es, action was taken for the complaint 


ACTION FOR CO«ff>LAINTS-(>C 


ACTION FOR COMPLAINTS 





(3 •) Mo such case was 
ftxirtd 

Kb) Vtts, but no action 
I ms taken 

oc) Yes, action was taken 
’ for the complaint i 



□ a) No such case was 
found 

■ b) Yes, but no action 
was taken 

□ c) Yes. action was taken 

for the complaint 


Figure 4.32 


Figure4.33 


b> but no 
action was 
taken 


SERVICES 


c) Yes, action 
was taken for 
the complaint 
11 % 


a) No such 
case was found 



80 % 


Figure 4.34 


A very large population, almost every body says that they never encountered such type of 
case but there is a dilfferent picture also which came in front through open ended question 
but that population is not big enough to establish any fact. 


“Does it possible to complain? ” 

— Student comment 

4.2.6.9 Staff 

OVERALL POINTS 3.03 

OVERALL RATING GOOD 
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Do you find the supporting staff- 
! ) Technically capable 

a) Yes 

b) Average 

c) No 


POINTS 3.19 
RATING GOOD 



Figure 4.35 Figure 4.36 


TECHNICAL CAPABILITY 



b) No 
4 % 


Figure 4.37 


Half of the students think that the supporting staff is technically capable. But there were 
some surprising comments regarding the supporting staff 

“Does the supporting staff exist?” 

~ Student comment 

Almost all the comments of this type were given by CC users only. 


apisrirus? 

-.,-1 


51 


2) Approachable 

a) Yes 

b) Not always 

c) No 


POINTS 2.88 
RATING AVERAGE 


APPROACHABLE-PG 





APPROACHABLE-UG 



Figure 4.38 


Figure 4.39 


APPROACHABLE 

! 


I 

{ 


c) Hot Always 
- 45 ^^ 



b) No 
15% 


Figure 4.40 


Almost a ‘good’ rating by students as far as approach to staff is concerned. 


3) Helping 

a) Yes 

b) Not always 

c) No 
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POINl'S 

RATINC; 


3.03 

G(X)D 



Figure 4.41 


Figure 4.42 


HELPING 


c) Not every 
time 
37 % 


b) No 
1 - 4 % 




a) Yes 
49 % 


Figure 4.43 


As the above charts depict in this area as well students find the supporting staff good. 


4.2.6.10 Security 
POINTS 2.61 

RATING AVERAGE 


Do you find the IT System secure? 

a) Absolutely secure 

b) Reasonably secure 

c) Not secure 

d) Dangerously vulnerable 
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Figure 4.44 


Figure 4.45 


SECURITY 


d> Oangerously 
vulnerable 
8% 


a) Absolutely 
secure 
4 % 


c> Not secure 
27% 



secure 

ei% 


Figure 4.46 

A good population of people thinks that the IT system of the IIT Kanpur is reasonably 
secure though overall it is rated as average. 

4.2.6.11 User Support 
OVERALL POINTS 2.71 

OVERALL RATING AVERAGE 

10.1) Is there any online/offline help available for the availability, accessibility of 
software packages? 

a) Yes 

b) Can’t say 

c) No 
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POINTS 

RATINO 


2.67 

AVERAGE 


ONtWEroFFUNE HELP.?G 


ONLINE/OFFLINE HELP-UG 



Figure 4.47 


Figure 4.48 


ONl_INE/OFFI_INE HELP 



to) No 
20% 


Figure 4,49 

As evident from the feed back that almost half of the people axe even do not know 
whether online/ offline help is available. It is equally bad as the help is not available. It 
again shows the fact which was established by the first question that students are not 
aware of the facilities provided by the IIT Kanpur. 

“A few digital tutorials on important and most frequently used software would be welcome” 

-- Student comment 

10.2) Were extra needs like more hard disc space, parallel computing, use of plotter etc taken care 
of whenever required? 

a) Yes 

b) Can’t say 

c) No 
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POINTS 

RATING 


2.6 

AVERAGE 


EXTRA NEEDS-PG 


1 EXTRA NEEDS-UG 

) 

I 



Figure 4.50 


Figure 4.51 


EXTRA NEEDS 



Figure 4.52 


In this area UG students and PG students differ considerably. Almost half of the UG 
students say they never had such kind of needs on the other hand almost same percentage 
of PG students say that Yes needs were taken care of. Reason may be that PG students 
require more such kind of things due to their specialized field of study or may be because 
of thesis work. 

Though overall rating of this area is average but it can be considered as good due to 
favorable response of PG students. 
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4.3 Some Other Statistics: Discussion Continued 

Some questions were asked just from the point of view of statistics. Responses to those 
questions are as follows: 

Do you have a PC in your room? 

a) Yes 

b) No 


PC IN ROOM 


a) Yes 



b> No 


Figure 4.53 


If YES, you bought it because 

a) PC was not always readily available 

b) Wanted to avail the facilities to the fullest 

c) Can’t say 

REASON FOR HAVING PC IN ROOM 

Can’t say Pq was not 



Figure 4.54 
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U'hat flaws do you find in the IT/ computing facilities in your lab/CC.? 

a) Software required for a particular task is not available or the older version is available 

b) Hardware is not compatible to the software 

c) (’oniputer is not available every time 

d) Accessibility of the software 

e) Training/ help manuals are not available 

f) Never knew that such kind of software is available 

g) Any other 


FLAWS-PG 



m a) Software required for a 
particular task is not available 
or the older version is available 
m b) Hardware Is not compatible 
to the software 

□ c) Computer is not available 
every time 

□ d) Accessibility of the software 


■ e) Training/ help manuals are 
not available 

B f) Never knew that such kind of 
software is available 


Figure 4.55 
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FLAWS-UG 


n a) Software required for a 
particular task is not available 
or the older version is available 


■ b) Hardware is not compatible 
to the software 


a.12% 



e.38% 


□ c) Computer is not available 
every time 

□ d) Accessibility of the software 


B e) Training/ help manuals are 
not available 


m f) Never knew that such kind of 
software is available 


Figure 4.56 
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Figure 4.59 


4,4,2 Main Responses 


MAIN COMMENTS 



1 . T raining is not available for software 

2. 24 hours internet facility should be made available in CC 

3. Internet speed is very slow 

4. IT-faciiities are best possible/sufficient 

5. People are not aware of available facilities/ There is a need of awareness program 

6. A large number of PCs does not work properly/Maintenance is poor 

7. More Hard disc space should be given to the students 

8. More software should be made available/ Required s/w not available 

9. Miss use of computing/ internet facilities is very high 
1 0 LAN speed is very slow 

1 1 . Online help should be made available 

12. Others 

Figure 4.60 
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4.4.3 Summary of Student Concerns 


COWMENT RESPONSES 

People are not aware of available fecilities/ There is a need of awareness program 27 

Internet speed is very slow 42 

More software should be made available/ Required s/w not available 18 

Online help should be msKie available 12 

LAN speed is very slow 15 

24 hours internet feciiity should be made available in CC 51 

IT facilities are best possible/sufficient 36 

RAM is less in many systems 3 

Many games are not available 3 

PC should be made available in rooms also 9 

Training is not available for softwares 57 

CC management team/ suf^rting staff is not helpftil/mde 9 

Printer is not accessible always specially on Sunday when it is nj^ded the most 6 

Improvement is needed 3 

Laser printing should be made cheaper 3 

Older version of softwrare is installed ® 

Help manuals are not helpfu l 

Miss use of computing/ internet feciiities is very high ’I® 

Students do not care of the feciiities provided ^ 

Facilities are not availed even to 50% of its capacity/ under uti[ization 6 

Sen«r down time is very high ® 

Systwn is not secure , _ . - ^ 

PCs should be upgraded ^ 

A large number of PCs does not wrk properly/Maintenanrre js poor 24 

More Hard disc space should be given to the students 1 5 

I do not know vrtiether supporting staff exists? ® 

Some times when one really needs a PC somebody else is playing games/ watching a movie 3 

CC should maintain a library of the books related to the software/ hardware which is in use 3 

Number of PCs should be increased in NT lab {CC)/Jn ariy other lab 9 

Mostly only demo version is available ^ 

Help manuals are not accessible . ^ 

No audio output in many machines ^ 

Laser printing facility should be m^e availabl e for 24 hou rs ,3 

installation penriission of software should be gh«n ® 

Space is not sufficient _ ^ 

Systems get hang^ very frequently ^ 

Networking is not proper in GH ^ 

Chatting shoi^ be alfow^ _ . . - ^ 

TOTAL NUMMR OF COMMENTS 


Table 4.5 Main Concerns Expressed by the Students 
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4.5 Administration Survey 

In this survey the target population was IT lab administrators. In some labs administrators 
were faculty nienihers while in some labs administrators were the people who are meant 
for that job only. This .survey was personal interview type. Questionnaire for this survey 
is given in appendix II. 

4.5.1 Survey Population 

There are around 100 computer labs in IIT Kanpur out of which 30 lab administrators 
were chosen randomly almost from every department including CC. The responses do not 
have much variation; almost every administrator said similar things. This survey is kept 
purely qualitative in nature. 

4.5.2 Results and Discussion 

In this section results are given in detail question by question. 

4.5.2.1 How many software/ packages do you have in your lab? Is their any list of 
software maintained for users? 

In answer to this question only a few said that they have been maintaining a list. 
Specifically only two departmental labs and CC have this kind of facility. 

This confirms the student opinion that there is no awareness program regarding IT 
facilities. Only CC conducts an awareness program every year for this but most of the 
students said that it is not sufficient [Section 4.2.6. 1]. No otiier lab provides such kind of 
awareness program. 

4.5.2.2 How a decision regarding the purchase of software or hardware takes place? 
Most of the administrators said that software is purchased on requirement basis and it is 
not a very complicated procedure to decide what to buy. As far as hardware is concerned 
it depends on a number of things like student- machine ratio, technology up gradation, 
budget etc. 
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4.5.2..^ Is there any record for the frequency of use of particular software? 

None ol the lab maintains such kind of record and most of them think that it is not 
required either. 

4.5.2.4 After the purchase of particular software do you provide some tr ainin g of 
that softw are to the students? 

Only a few labs provide some initial training which is most of the times provided by the 
firm troin which the software is purchased. Training manuals are available in most of the 
labs but as it is evident from student survey that training manuals are not helpful most of 
the times. 

One interesting point was raised by some of the people that students are not interested in 
training. A number of times tutorials have been arranged but a very few students turned 
up. 

4.5.2.5 How many software are found redundant? What is the reason of 
redundancy? Is it because of 

a) Accessibility 

b) Lack of proper training 

c) Ambiguously written help manuals 

d) Poor support by the software firm 

c) Software is similar to any other software 

f) Updated version is not available 

g) Hardware compatibility 

h) Any other 

In most of the labs no such case was found. In a couple of labs there were such cases but 
they all had same reason that the software they purchased went free later on. 

4.5.2.6 What kind of contract do you have with the software company? Commercial, 
educational etc. 

Almost every lab buys educational version of the software. Some administrators also said 
that some times using pirated versions as far as trial is concerned. 
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4.5.2.7 Arc there any compiaints that due to unavailabiKty of particular software a 
certain task could not be carried out? If yes then what was the reason of 
unavailability? What action was taken? 

Normally some action was immediately taken if the requirement was genuine but a 
number of times requirements were not found genuine. Though in most of the labs there 
is no formal feedback system for the users and verbal feedback usually takes place. 

4.5.2.7 How the budget is allocated? 

Budget allocation depends on the previous year’s budget as well as the current year’s 
projections. Money also comes from some other sources like consultancy and commercial 
projects. Most of the administrators said that money normally is not a big problem. 

4.5.2.8 Is the IT system secure? 

Most of the administrators said the system can not be made totally secure. Students are 
made aware of the security issues time to time, but some students intentionally use the 
channels which are insecure or are indulge in activities like hacking. They said it is more 
of moral responsibility issue then the technical issue, 

4.5.2.9 Any thing else you would like to share? 

A majority of the population did not share their views but a good number of people 
showed their concerns towards the miss use of the system. Some people also said that the 
facilities are not uniform through out the IIT for example in some departments laser 
printing is free while some others charge money which is less then the CC printer. This is 
causing problem. 
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Chapter 5 


Conclusion and Suggestions 


5.1 Ranking 

Based on the results of the study the identified areas are ranked which represents their 
performance and contribution to the IT system of IIT Kanpur. Ranks are given in Table 
5.1. 


RANK AREA OF STUDY 

! internet and Networking 

3 Hardware Facilities 

'=r 4 ■ ' T1mbSp%' ' 

5 • Staff 


4 Awaien^^ 

7 User Support 



9 Services 


;,|f V.1 Utiiizattqn;,^ ' ■ 

1 1 Training 

Table 5.1 Ranking 

5,2 Strengths of the IT System 

Following are the strengths of the IT system of IIT Kanpur- 

> Internet and Networking facilities 

> Software Facilities 

> Hardware Facilities 
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I Iscrs are quite satisfied with these facilities. These areas are working effectively and 
eftlcieiitiy. Point of concern is only that these facilities are directly or indirectly 
dependent on the other areas. Poor performance of other areas may cause harm to the 
performance of these facilities. 

5.3 Weaknesses of the IT System 

Following are the weaknesses of the IT system of IIT Kanpur- 
> Training Facility 
V Utilization 

Users are not at all satisfied with the training facility provided for the computing work. In 
fact there is nothing like providing training in any of the labs. Both students and 
administration think that there is a very heavy miss use of facilities as well as the 
facilities axe under utilized. 

These two areas need some special attention of the administration and awareness among 
students. 

5.5 Action Points (Priority wise) 


Following action points are suggested to improve efficiency and effectiveness of the 
system: 

1. Development of a web page with involvement of each and every department. It 
should contain the list of software, packages and machines and other hardware available 
in respective departments. This web page should also contain a few tutorials of the most 
frequently used software. It will help in overcoming the problems of training, awareness 
and underutilization. 

2. Training should be provided for the software most frequently used on departmental 
level. 

3. Miss use of the facilities should be checked. This can be done by more involvement 
of administration and through awareness among the students. 
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4. A help desk should be provided in CC. A feedback system should be established 
through this help desk. For departmental labs also feedback system should exist. It will 
help in improving user support. 

5. A lihrars' maintaining help manuals and related books should be established in CC. 

(>. Facilities through out the institute should be made consistent. 

7. In some labs pirated softwares are used, this practice is unethical and should be 
avoided. 

H. Filtering of net access should be made more effective. 
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Appendices 


I. Student Survey 


Evaluation of IT scenario in IIT Kanpur 
Questionnaire for the survey 

UG/PG/PHD Branch . Department lab/CC/Both 

NOTE: You can tick my number of options. 

1) Are you aware how many software/ packages do you have in your lab or in CC? 

a) Fully aware 

b) Not concerned 

c) Aware about whatever is required 

2) Was there any orientation program held in the beginning for the awareness of IT 
facilities available in IIT Kanpur or at least in your lab? 

a) Yes 

b) No 

c) Yes, there was a program but it was not sufficient d) Not aware 

3) Which software/ package you use most frequently? (Preference wise) 

1 ) - 

2 ) 

3) 

4) Is there any training facility available for the software you worked on? 

a) Yes 

b) No 

c) Yes but not sufficient 

d) Training is not required 
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5) If the training facility was not available then what was the estimated fraction of time 
spent on non productive work like figuring out which software to use, how to access it 
and how to use it? 

a) 25 percent 

h) Between 25 percent to 50 percent 

c) Between 25 percent to 50 percent 

d) > 75 percent 

6) What flaws do you find in the IT/ computing facilities in your lab/CC.? 

a) Software required for a particular task is not available or the older version is available 

b) Hardware is not compatible to the software 

c) Computer is not available every time 

d) Accessibility of the software 

e) I'raining/ help manuals are not available 

f) Never knew that such kind of software is available 

g) Any other 

7) Are IT facilities available in IIT Kanpur sufficient? 

1) Hardware facilities 

a) Yes 

b) No 

c) Yes, but there is a room for improvement (please mention what improvement do you 

want .) 

2) Software/packages available 

a) Yes 

b) No 

c) Yes, but there is a room for improvement (please mention what improvement do you 

want ) 

3) Net connection and other networking facilities 

a) Yes 

b) No 

c) Yes, but there is a room for improvement (please mention what improvement you 

want ) 
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8) Arc students misusing the IT facilities available in IIT Kanpur? 
u) Yes 
h) No 

cK'an't sijy 

‘)) Il'Yl-S, what resources are misused? 

a) Printer, scanner, cd writer etc 

b) Net facilities 

c) I lard disk space 

1 1)) Are students under utilizing the IT facilities available in IIT Kanpur? 

a) Yes 

b) No 

c) Can’t say 

11) Have you ever complained to the concerned person regarding the unavtdlability or 
inaccessibility of a particular software? 

a) No such case was found 

b) Yes, but no action was taken 

c) Yes, action was taken for the complaint 

12) How do you find the supporting staff? 

1) Technically capable: 

a) Yes 

b) No 

c) Average 

2) Approachable: 

a) Yes 

b) No 

c) Not Always 

3) Helping: 

a) Yes 

b) No 

c) Not every time 

13) Do you have a PC in your room? 
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a) \'cs 
h) No 

14) If VI'S, \ou bought it because 

a) PC’ \va.s not always readily available 

b) Wanted to a\ ail the facilities to the fullest 

c) Can't say 

15) is there any online/offiine help available for the availability, accessibility of software 
packages? 

a) Yes 

b) No 

c) C^an’t .say 

16) Were extra needs like more hard disc space, parallel computing, use of plotter etc 
taken care of whenever required? 

a) Yes 

b) No 

c) Some times 

d) I never had such kind of needs 

17) Do you find the IT System secure? 

a) Absolutely secure 

b) Reasonably secure 

c) Not secure 

d) Dangerously vulnerable 

1 8) Any thing else you would like to share regarding IT facilities in IIT Kanpur. 
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II. Administration Survey Questionnaire 


Evaluation of the IT Scenario in IIT Kanpur 
Questionnaire for the survey 

ADMININSTRATION 

1 ) How many software/ packages do you have? Is their any list of software maintained 
for users? 

2) How a decision regarding the purchase of software takes place? 

3) Is there any record for the frequency of use of particular software? 

4) After the purchase of particular software do you provide some training of that 
software to the students? 

5) How many software are found redundant? What is the reason of redundancy? Is it 
because of 

a) Accessibility 

b) Lack of proper training 

c) Ambiguously written help manuals 

d) Poor support by the software firm 

e) Software is similar to any other software 

f) Updated version is not available 

g) Hardware compatibility 

h) Any other 

6) What kind of contract do you have with the software company? (Commercial; 
educational etc). 

7) Are there any complaints that due to unavailability of particular software a certain 

task could not be carried out? If yes then what was the reason of unavailability? 
What action was taken? 

8) How the budget is allocated? 

9) Is the IT system secure? 

10) Any thing else you would like to share? 
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